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LONG-TERM  GOALS 

This  project  seeks  to  develop  the  technology  to  fuse  high  spatial  resolution  MultiSpectral  Imagery 
(MSI)  with  lower  spatial  resolution,  but  higher  spectral  resolution,  HyperSpectral  Imagery  (HSI)  to 
provided  enhanced  target  detection  and  battlespace  characterization.  It  seeks  to  push  beyond 
traditional  PAN  sharpening  techniques  to  develop  applications  that  will  increase  the  accuracy  and 
fidelity  of  the  sharpened  spectral  imagery. 

OBJECTIVES 

1)  Co-register  high  resolution  MSI  data  with  HSI  data  collected  during  the  Fall  of  2004. 

2)  Correct  the  MSI  data  for  illumination  and  atmospheric  interference  to  derive  a  broadband  remote 
sensing  reflectance,  Rrs,  measurement  from  the  MSI  data. 

3)  Develop  methodologies  to  fuse  the  HSI  data  with  the  higher  spatial  resolution  MSI  data,  using  the 
spectral  characterization  of  the  MSI  Rrs  data  stream. 

APPROACH 

The  fusion  of  remote  sensing  data  from  different  sensors  has  a  long  history  of  success  in  the  terrestrial 
environment  (Pohl  and  Genderen,  1998).  Its  application  in  ocean  remote  sensing  has  accelerated  in 
recent  years  with  the  use  of  multiple  resolution,  multiple  frequency  optical  and  microwave  imaging. 
The  optical  imagers  include  both  active  (e.g.  LIDAR)  and  passive  (e.g.  HyperSpectral  Imagers,  HSI). 
Microwave  imagers  are  typically  Synthetic  Aperture  Radar  (SAR).  While  the  field  has  been  generally 
confined  to  airborne  and  space-based  platforms,  imagery  data  fusion  may  also  include  some 
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combination  of  data  from  these  platfonns  with  data  from  in-water  systems  such  as  multibeam  and  side- 
scan  sonar  imagery.  In  short,  data  fusion  is  the  process  by  which  two  or  more  streams  of  data  of 
different  temporal  and  spatial  resolution  are  combined  to  produce  synthesized  products  which  are 
unattainable  from  a  discrete  image. 

The  advantages  of  data  fusion  are  fairly  clear.  In  the  temporal  domain,  data  fusion  allows  for  the  more 
complete  imaging  of  an  area  intermittently  covered  by  clouds.  It  may  also  provide  better  analysis  of 
movement  or  time-dependent  change  in  targets  and/or  environmental  conditions.  In  the 
electromagnetic  frequency  domain,  targets  which  are  visible  in  one  image  type  may  be  less  than  visible 
in  another,  and  vice  versa.  SAR  imagery  may  illuminate  target  elevations  or  directional  wave  spectra, 
whereas  optical  imagery  may  identify  targets  by  shape  and  color.  Combining  the  targeting  and 
identification  of  these  different  data  types  provides  target  recognition  and  environmental 
characterization  with  high  confidence  and  lower  false  alarm  rates  than  may  be  found  from  single 
source  imagery.  In  the  spatial  domain,  objects  that  are  identifiable  on  the  basis  of  size  and  shape  in 
panchromatic  (PAN)  imagery  are  often  unidentifiable  by  their  geometric  dimensions  in  lower 
resolution  imagery  multispectral  or  hyperspectral  imagery.  The  converse  is  that  the  spectral  imagery 
provides  the  ability  to  identify  targets  or  characterize  the  environment  based  on  unique  spectral 
signatures,  which  is  unavailable  in  the  panchromatic  imagery,  but  may  not  be  able  to  resolve  the  shape 
or  dimensions  of  small  targets. 

This  project  seeks  to  co-locate  simultaneously  collected  MSI/HSI  data  streams  to  develop  enhance 
imagery  sharpening  capabilities  for  use  in  target  detection  and  environmental  characterization.  It  seeks 
to  build  upon  PAN  sharpening  techniques  by  using  the  full  spectral  information  provided  by  MSI  data 
stream.  We  hypothesize  that  the  best  way  to  attempt  this  image  fusion  is  at  the  level  of  remote  sensing 
reflectance,  so  the  MSI  data  must  be  corrected  for  illumination  and  atmospheric  interference.  The 
spectral  information  from  the  co-registered  MSI/HSI  Rrs  data  will  then  be  fused  to  provide  more 
accurately  sharpened  HSI  data  for  spectral  target  algorithms. 

WORK  COMPLETED 

The  work  during  this  period  looks  at  exploiting  the  information  available  from  multiple  sensors  aboard 
an  aircraft.  Particularly,  we  wish  to  combine  infonnation  from  a  high  spatial  but  low  spectral 
resolution  camera  and  a  low  spatial  but  high  spectral  resolution  camera.  The  desired  final  image 
should  have  both,  high  spatial  and  high  spectral  resolution.  Data  fusion  can  be  defined  as  the  fusion  of 
complementary  infonnation  from  multiple  sources.  In  this  work,  we  focus  on  the  combination  of  low- 
resolution  multi-spectral  data  with  high-resolution  tri-band  (RGB)  data.  Many  image  fusion 
techniques  have  been  proposed  to  merge  multi-spectral  (MS)  and  panchromatic  (Pan)  images,  with 
complementary  characteristics  of  spatial  and  spectral  resolutions.  They  are  mostly  component 
substitution  based  methods  such  as  IHS,  Brovey  and  MULTI. 

The  IHS  (Intensity,  Hue,  and  Saturation)  transformation  is  mostly  used  for  three  band  images.  In  the 
IHS  transfonnation,  the  difference  between  the  high  resolution  and  low  resolution  intensity  images  is 
added  to  the  up-sampled  MS  bands.  However,  as  the  spectral  response  of  the  two  cameras  is  usually 
different,  this  procedure  is  prone  to  spectral  distortion.  When  more  than  three  bands  are  present,  this 
transformation  can  be  applied  to  groups  of  three  consecutive  bands  at  a  time.  Alternatively,  a  PC  A 
(Principal  Component  Analysis)  approach  can  be  used. 
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The  Brovey  fusion  method  computes  the  ratio  between  a  panchromatic  cell  value  and  the  average  of 
the  corresponding  multi-spectral  cell  values  and  uses  that  ratio  to  compute  the  final  color  component 
values  for  the  pan-sharpened  image.  The  Gram-Schmidt  (GS)  technique  is  another  component 
substitution  technique  for  spectral  sharpening..  In  the  GS  method,  the  low-resolution  version  of  the 
PAN  image  is  obtained  as  the  pixel  average  of  the  MS  bands.  However,  as  discussed  earlier,  because 
of  the  radiometric  differences  between  the  two  sensors,  the  reconstructed  image  is  prone  to  suffer  from 
spectral  distortion.  The  purpose  of  this  work  is  to  mitigate  this  distortion  by  reconciling  the  two 
sensors  radiometrically,  prior  to  perfonning  the  GS  fusion.  A  similar  approach  is  taken  in  Aiazzi  et  al, 
2006  where  regression  coefficients  are  determined  to  reconcile  the  two  sensors  prior  to  GS  sharpening. 
In  our  case,  however,  the  sensor  response  is  known  a  priori,  obviating  the  need  for  determining 
coefficients. 

RESULTS 

The  purpose  of  this  work  is  to  construct  a  high  spatial  and  high  spectral  resolution  image  using  images 
from  two  sensors.  One  sensor  is  a  high  spatial  resolution  frame  camera  -  the  DSS-322.  The  other  is  a 
high  spectral  resolution  camera,  FERI’s  SAMSON.  The  Gram  Schmidt  procedure  described  above 
was  used  to  combine  information  from  the  two  cameras  to  obtain  a  high  spectral,  high  spatial 
resolution  image. 

To  evaluate  the  validity  of  the  reconstructed  image,  the  hyper-spectral  and  multi-spectral  images  had  to 
be  down-sampled  spatially  and  then  used  to  reconstruct  the  original  hyper-spectral  image.  The  error 
between  the  reconstructed  and  original  hyper-spectral  image  is  a  quantitative  indicator  of  the  proposed 
procedure’s  validity. 

Image  fusion  was  studied  under  two  conditions  -  practical  and  ideal.  In  the  practical  scenario,  actual 
hyper-spectral  and  multi-spectral  images  were  used.  For  the  ideal  scenario,  an  actual  hyper-spectral 
and  a  simulated  multi-spectral  image  (simulated  using  the  hyper-spectral  image)  were  used.  By  using 
hyper-spectral  and  multi-spectral  images  derived  from  the  same  sensor,  many  error  sources  such  as 
registration,  calibration  and  atmosphere  are  eliminated.  Comparing  results  from  these  two  scenarios 
helps  quantify  the  error  contribution  from  these  sources. 

The  simulated  DSS  image  was  obtained  from  the  hyper-spectral  image  using  the  spectral  response 
function  of  the  DSS  camera.  Each  band  of  the  DSS  camera  was  simulated  by  taking  a  dot  product 
between  that  band’s  spectral  response  function  with  the  actual  wavelength  responses  from  the  hyper- 
spectral  signal  as  shown  in  Equation  1 . 


DSS(i,j,k)=  HYP(i,j,  X)  x  SRFk  (T)  (1) 

^='lmin 

Here,  k  indicates  the  DSS’s  band,  k=  { 1,2,3},  (i,j)  indicate  the  pixel  location,  SRFk(A )  is  the  kth  band’s 
response  at  wavelength  A  . 

Two  approaches  to  image  fusion  were  compared.  In  the  first  approach,  GS  was  run  using  the  hyper- 
spectral  image  and  one  of  the  DSS  bands  to  obtain  the  reconstructed  hyper-spectral  image  as  shown  in 
Figure  1. 
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Figure  1.  In  the  traditional  Gram-Schmidt  approach  a  single  PAN  channel  is  used  to 
fuse  the  high  spatial  resolution  with  the  lower  spatial, 
but  high  spectral  resolution  image. 


In  the  second  approach,  the  hyper-spectral  image  was  multiplied  by  each  band’s  SRF  response,  to 
obtain  three  ‘tinted’  hyper-spectral  images.  Three  reconstructions  were  performed  to  obtain  ‘tinted’ 
reconstructed  images.  They  were  then  multiplied  by  the  corresponding  inverse  SRF  response  to  obtain 
three  versions  of  the  original  hyper-spectral  image.  Figure  2  illustrates  how  a  hyper-spectral  image 
was  generated  using  the  red  channel. 
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Figure  2.  In  the  modified  Gram-Schmidt  approach  each  RGB  channel  is  used  only 
with  the  overlapping  bands  in  the  HSI  image  cube. 
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Tinting  the  image  prior  to  using  the  GS,  ensures  that  the  hyper-spectral  and  multi-spectral  images  have 
similar  spectral  content  leading  to  better  reconstruction.  Once  three  versions  of  the  reconstructed 
hyper-spectral  image  are  obtained,  they  can  be  averaged  to  obtain  a  final  reconstructed  version.  In  this 
work,  a  more  elegant  fusion  scheme  was  used  as  shown  in  Equation  2. 

HYPavg(  ^  )=Wr(  ^  )HYPr(  ^  )+Wg(  ^  )HYPg(  ^  )+Wb(  ^  )HYPb(  ^  )  (2) 

Where  Wr(  ^  )  =  SRFr(  ^  )  /  (SRFr(  ^  )  +  SRFg(  ^  )  +  SRFb(  ^  )).  This  means  that  instead  of  weighting 
the  three  images  equally  (as  you  would  in  an  average),  they  are  now  weighted  in  proportion  to  their 
dominance  at  each  wavelength. 

DISCUSSION 

In  this  section,  we  compare  the  perfonnance  of  the  modified  method  to  that  of  the  original  method  in 
the  ideal  scenario.  Then,  we  compare  our  method’s  performance  in  the  two  scenarios  (ideal  and 
practical).  This  is  followed  by  a  detailed  discussion  of  the  results. 

First  we  evaluate  the  efficacies  of  the  proposed  modified  method  and  the  original  method  in  the  ideal 
scenario.  Figure  1  and  Figure  2  described  the  workings  of  the  two  methods.  A  three  band  multi- 
spectral  image  was  simulated  from  the  hyper-spectral  image  as  described  by  Equation  1 .  The  original 
hyperspectral  image  was  reconstructed  using  both  methods  and  the  errors  were  evaluated. 

To  characterize  the  nature  of  the  reconstruction  error,  we  studied  its  distribution  along  wavelengths 
and  magnitudes.  The  error  is  expressed  as  a  fraction  of  the  original  hyper-spectral  image. 

Figure  3  shows  the  reconstruction  errors  in  the  ideal  scenario  using  both  methods.  The  X-axis  ranges 
from  band  #40  to  band  #80  spanning  wavelengths  from  400nm  to  700nm.  The  Y-axis  expresses  the 
error  as  a  fraction  of  the  original  value.  From  (a),  it  is  evident  that  the  reconstruction  error  using 
Method  1  is  in  the  15-20%  range.  From  (b),  it  can  be  seen  that  using  the  Tinting  method,  the  error 
drops  to  the  6-8%  range. 
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Figure  3.  The  graph  on  the  left  shows  the  errors  between  in  the  ideal  case  use  the 
traditional  Gram-Schmidt  approach,  range  is  -15-20%  over  the  spectrum.  In  the 
modified  Gram-Schmidt  approach  (right  graph)  there  is  a  marked  reduction  in  the 
reconstruction  error,  range  is  ~6-8%  over  the  spectrum. 


Next  we  compare  the  reconstruction  errors  in  two  different  scenarios  using  the  Tinting  method.  Figure 
4  compares  the  errors  in  the  ideal  and  practical  scenarios. 


Figure  4.  This  graph  shows  the  reconstruction  error  in  the  practical  case,  where  the 
DSS  imagery  is  used  in  the  modified  Gram-Schmidt  approach  to  fusing  the  MSI/HSI 
data.  The  increase  in  the  error  range  to  -22%  results  from  differences  in  (1)  the 
unknown  calibration  of  the  DSS,  (2)  view  angle  and  its  impacts  on  atmospheric  and 
illumination  contamination,  and  (3)  pixel  co-registration  errors. 
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From  Figure  4,  it  is  evident  that  as  we  move  from  the  Ideal  to  the  Practical  scenario,  the  average  error 
jumps  from  about  5%  to  22.5%.  There  are  multiple  reasons  for  this  jump  in  reconstruction  error  as  we 
move  from  the  Ideal  to  Practical  scenarios. 

First,  errors  arise  because  the  multi-spectral  camera  is  not  accurately  calibrated.  We  have  sought  to 
work  with  the  manufacturer  to  resolve  these  calibration  issues,  but  as  of  yet,  this  sensor  can  not  be 
calibrated  to  any  radiometric  standard. 

Second,  the  two  sensors  look  at  the  same  location  through  different  view  angles,  the  hyper-spectral 
camera  view  is  nominally  at  a  nadir  angle  in  the  along  track  direction,  and  — F/-20  degrees  in  the  cross 
track  direction  (pushbroom  type  sensor),  whereas  the  multi-spectral  camera  usually  looks  at  an  angle 
varying  from  0-20  degrees  in  both  the  along  track  and  cross  track  directions  (framing  type  sensor). 
Thus  their  path  lengths  through  the  atmosphere  and  each  pixels  relative  illumination  are  different, 
which  yield  a  different  overall  spectral  signal  between  the  DSS  and  SAMSON  when  the  data  are  not 
corrected  for  atmospheric  and  illumination  differences. 

Third,  since  two  cameras  have  different  resolution  and  lens  aberration,  image  registration  is  inaccurate. 
It  was  observed  that  a  registration  error  of  one  pixel  can  contribute  up  to  5%  error. 

We  hypothesize  that  of  the  20%  average  error,  errors  due  to  atmosphere  and  calibration  account  for 
10%  of  the  error,  5%  due  to  image  mis-registration  and  the  remaining  5%  is  inherent  in  the  GS 
procedure.  In  spite  of  the  errors,  this  approach  shows  great  promise  in  being  able  to  generate  HSI  data 
at  the  spatial  resolution  of  PAN  and  RGB  imagers.  We  need  to  cross  the  critical  step  of  calibrated  the 
RGB/MSI  data  so  that  we  may  apply  this  technique  to  the  reflectance  data  itself,  rather  than  the  sensor 
radiance  data.  This  would  greatly  reduce  the  errors  in  the  Practical  Approach  shown  here,  allowing  us 
to  approach  our  5%  reconstruction  error  seen  in  the  Ideal  Approach. 

IMPACTS/APPLICATIONS 

The  abilities  to  detect  targets  and  characterize  the  environment  in  the  Very  Shallow  Water  (VSW)  and 
beach  zone  are  critical  to  MIW,  MCM,  and  NSW  operations.  The  image  fusion  techniques  here  will 
provide  enhanced  abilities  to  detect  and  characterize  target  in  access  denied  areas  using  both 
geomorphology  (shape)  and  spectral  (color)  signatures.  In  addition,  the  successful  development  of 
these  techniques  will  feed  back  into  the  development  of  new  imaging  systems  for  space  and  aircraft 
platforms,  including  Unmanned  Aircraft  Systems  (UAS). 

RELATED  PROJECTS 

This  project  works  closely  with  those  of  Pis  Bissett  and  Kohler,  including  N000 1401 10201, 

N000 1 403 1 0626,  and  N000 1 404 1 0297 
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